Assembly line with integrated electronic visual inspection

ABSTRACT

Methods and systems are disclosed for obtaining a first image of a tray, determining a presence or absence of one or more first patterns in the first image, determining a rotation of each the one or more first patterns in the first image, and performing an action based on the presence or absence and the rotation of the one or more first patterns in the first image.

This application is the U.S. national phase entry under 35 U.S.C. § 371of International Application No. PCT/US2017/043595, filed Jul. 25, 2017,which claims priority to U.S. Provisional Application No. 62/368,438,filed on Jul. 29, 2016, the entirety of which is incorporated byreference herein.

BACKGROUND

Assembly of products that contain multiple items, such as pharmaceuticalpackaging, is a complex task. The assembly can proceed in one or morestages with items being placed into the product at each stage. Errorscan be introduced at each stage by failing to place a correct item inthe product, placing too many of the correct item in the product, and/orplacing an incorrect item in the product. Products that are ultimatelyshipped with errors result in lost revenue, increased customercomplaints, and lost time in addressing the customer complaints. In thecase of a pharmaceutical product package, one unintended result ofimproper packaging is that clinicians or patients may be unwilling touse a pharmaceutical product contained within an improperly assembledpackage. This can be particularly true for pharmaceutical products thatare administered parenterally, e.g., subcutaneously, intramuscularly,intravenously, intra-ocularly, or by inhalation. Even if an improperlyassembled package is returned to the manufacturer by a clinician or apatient, a regulatory agency, such as the U.S. Food and DrugAdministration, will not allow the pharmaceutical product to berepackaged, resulting in a Notice of Event (NOE). Such NOE's triggerinvestigations, added expense, and potentially result in an impairedcompetitive.

It would be desirable, therefore, to develop new technologies forproduct assembly, that overcomes these and other limitations of theprior art, and enhances it by reducing errors and increasing efficiencyof package assembly.

SUMMARY

It is to be understood that both the following general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive. Methods and systems are disclosed for obtaininga first image of a tray, determining a presence or absence of one ormore first patterns in the first image, determining a rotation of eachthe one or more first patterns in the first image, and performing anaction based on the presence or absence and the rotation of the one ormore first patterns in the first image.

Additional advantages will be set forth in part in the description whichfollows or may be learned by practice. The advantages will be realizedand attained by means of the elements and combinations particularlypointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate embodiments and together with thedescription, serve to explain the principles of the methods and systems:

FIG. 1 is an example system;

FIG. 2 is an example image of an object;

FIG. 3A is an example image of an object;

FIG. 3B is an example image of an object;

FIG. 4A is an example image of an object;

FIG. 4B is an example image of an object;

FIG. 5A is an example image of an object;

FIG. 5B is an example image of an object;

FIG. 6A is an example image of an object;

FIG. 6B is an example image of an object;

FIG. 7A is an example image of an object;

FIG. 7B is an example image of an object;

FIG. 8A is an example image of an object;

FIG. 8B is an example image of an object;

FIG. 9 is an example embodiment of an example system;

FIG. 10 is a flowchart illustrating an example method; and

FIG. 11 is an exemplary operating environment.

DETAILED DESCRIPTION

Before the present methods and systems are disclosed and described, itis to be understood that the methods and systems are not limited tospecific methods, specific components, or to particular implementations.It is also to be understood that the terminology used herein is for thepurpose of describing particular embodiments only and is not intended tobe limiting.

As used in the specification and the appended claims, the singular forms“a,” “an” and “the” include plural referents unless the context clearlydictates otherwise. Ranges may be expressed herein as from “about” oneparticular value, and/or to “about” another particular value. When sucha range is expressed, another embodiment includes from the oneparticular value and/or to the other particular value. Similarly, whenvalues are expressed as approximations, by use of the antecedent“about,” it will be understood that the particular value forms anotherembodiment. It will be further understood that the endpoints of each ofthe ranges are significant both in relation to the other endpoint, andindependently of the other endpoint.

“Optional” or “optionally” means that the subsequently described eventor circumstance may or may not occur, and that the description includesinstances where said event or circumstance occurs and instances where itdoes not.

Throughout the description and claims of this specification, the word“comprise” and variations of the word, such as “comprising” and“comprises,” means “including but not limited to,” and is not intendedto exclude, for example, other components, integers or steps.“Exemplary” means “an example of” and is not intended to convey anindication of a preferred or ideal embodiment. “Such as” is not used ina restrictive sense, but for explanatory purposes.

Disclosed are components that can be used to perform the disclosedmethods and systems. These and other components are disclosed herein,and it is understood that when combinations, subsets, interactions,groups, etc. of these components are disclosed that while specificreference of each various individual and collective combinations andpermutation of these may not be explicitly disclosed, each isspecifically contemplated and described herein, for all methods andsystems. This applies to all aspects of this application including, butnot limited to, steps in disclosed methods. Thus, if there are a varietyof additional steps that can be performed it is understood that each ofthese additional steps can be performed with any specific embodiment orcombination of embodiments of the disclosed methods.

The present methods and systems may be understood more readily byreference to the following detailed description of preferred embodimentsand the examples included therein and to the Figures and their previousand following description.

As will be appreciated by one skilled in the art, the methods andsystems may take the form of an entirely hardware embodiment, anentirely software embodiment, or an embodiment combining software andhardware aspects. Furthermore, the methods and systems may take the formof a computer program product on a computer-readable storage mediumhaving computer-readable program instructions (e.g., computer software)embodied in the storage medium. More particularly, the present methodsand systems may take the form of web-implemented computer software. Anysuitable computer-readable storage medium may be utilized including harddisks, CD-ROMs, optical storage devices, or magnetic storage devices.

Embodiments of the methods and systems are described below withreference to block diagrams and flowchart illustrations of methods,systems, apparatuses and computer program products. It will beunderstood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, respectively, can be implemented by computerprogram instructions. These computer program instructions may be loadedonto a general purpose computer, special purpose computer, or otherprogrammable data processing apparatus to produce a machine, such thatthe instructions which execute on the computer or other programmabledata processing apparatus create a means for implementing the functionsspecified in the flowchart block or blocks.

These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including computer-readableinstructions for implementing the function specified in the flowchartblock or blocks. The computer program instructions may also be loadedonto a computer or other programmable data processing apparatus to causea series of operational steps to be performed on the computer or otherprogrammable apparatus to produce a computer-implemented process suchthat the instructions that execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

Accordingly, blocks of the block diagrams and flowchart illustrationssupport combinations of means for performing the specified functions,combinations of steps for performing the specified functions and programinstruction means for performing the specified functions. It will alsobe understood that each block of the block diagrams and flowchartillustrations, and combinations of blocks in the block diagrams andflowchart illustrations, can be implemented by special purposehardware-based computer systems that perform the specified functions orsteps, or combinations of special purpose hardware and computerinstructions.

The present disclosure relates to improvements in computer functionalityrelated to manufacturing and product assembly.

FIG. 1 is a block diagram illustrating various aspects of an exemplarysystem 100 in which the present methods and systems can operate. Oneskilled in the art will appreciate that provided herein is a functionaldescription and that the respective functions can be performed bysoftware, hardware, or a combination of software and hardware.

In one aspect, the system 100 can comprise a conveyor belt 101. Theconveyor belt 101 can comprise one or more cleats 102. The one or morecleats 102 can be made of rubber or similar material for attachment tothe conveyor belt 101. The one or more cleats 102 can be raised orotherwise extend above the surface of the conveyor belt 101. The one ormore cleats 102 can comprise a leading cleat and a trailing cleat basedon a direction of travel 103. The leading cleat and the trailing cleatcan be relative to an object placed on the belt, such that the leadingcleat is in front of the object relative to the direction of travel 103and the trailing cleat is behind the object relative to the direction oftravel 103. Accordingly, a leading cleat for a first object can also bea trailing cleat for a second object that is ahead of the first objectand so on. One or more objects 104 can be placed on the conveyor belt101. In an aspect, the one or more objects 104 can comprise a product inone or more states of assembly. For example, the one or more objects 104can comprise a tray. The tray can be configured to hold one or moreitems. The one or more items can be related to a medical treatment. Forexample, the one or more items can comprise one or more syringes, autoinjectors, one or more syringe needles, one or more containers of amedicament, one or more pamphlets or sets of written instructions,combinations thereof, and the like.

In one aspect, the set of written instructions sets forth informationabout how to use and administer a medicament. In another aspect, thewritten instructions are a medication label approved by a regulatoryagency, such as the U.S. Food and Drug Administration.

In one aspect, the medicament is a solid formulation. In another aspectthe medicament is a liquid formulation. In another aspect the medicamentis a gel formulation.

In one aspect, the medicament is formulated for oral administration. Inanother aspect the medicament is formulated for parenteraladministration. In another aspect the medicament is formulated forsubcutaneous administration. In another aspect the medicament isformulated for intramuscular administration. In another aspect themedicament is formulated for intravenous administration. In anotheraspect the medicament is formulated for inhalation administration. Inanother aspect the medicament is formulated for intraocularadministration.

In one aspect, the medicament comprises a small molecule activeingredient. In another aspect, the medicament comprises a biologic. Inanother aspect, the medicament comprises a peptide or polypeptide activeingredient.

In one aspect, the medicament comprises a vascular endothelial growthfactor (VEGF) derivative active ingredient. In another aspect, themedicament comprises aflibercept, which is described in one or more ofU.S. Pat. Nos. 7,070,959; 7,303,746; 7,303,747; 7,306,799; 7,374,757;7,374,758; 7,531,173; 7,608,261; 7,972,598; 8,029,791; 8,092,803;8,343,737; 8,647,842, each of which is incorporated by reference in itsentirety.

The conveyor belt 101 can pass over a drive roll which can be driven bya stepper motor 105. The use of the stepper motor 105 enables precisepositioning of the one or more objects 104 relative to a camera 106, acamera 107, and a camera 108. The length of each of the one or moreobjects 104 can be represented as a precise number of motor steps. Theconveyor belt 101 can be precisely advanced or reversed to cause each ofthe one or more objects 104 to be moved into a field of view 109, afield of view 110, and a field of view 111, associated with the camera106, the camera 107, and the camera 108, respectively. A programmablelogic controller (PLC) 112 (the PLC 112 can comprise a computing device,a PLC, or other controller/processor) can be configured to cause thestepper motor 105 to execute any number of steps in either direction tocause the one or more objects 104 to be moved into the field of view109, the field of view 110, and the field of view 111.

In an aspect, the camera 106, the camera 107, and/or the camera 108 canbe configured for scanning, decoding, reading, sensing, imaging,capturing, and/or interpreting visual codes. In some aspects, the camera106, the camera 107, and/or the camera 108 can be configured to processlaser, linear, or area imaging. For example, in one aspect, the camera106, the camera 107, and/or the camera 108 may include an imager forscanning, reading, and decoding one-dimensional or two-dimensionalbarcodes. The camera 106, the camera 107, and/or the camera 108 caninclude any imager, barcode scanner, or visual code scanner capable ofextracting information from visual codes consistent with the disclosedembodiments. In certain aspects, the camera 106, the camera 107, and/orthe camera 108 can be configured to process scanned barcodes, images,and other data. The camera 106, the camera 107, and/or the camera 108can include one or more depth cameras for capturing, processing,sensing, observing, modeling, detecting, and interacting withthree-dimensional environments. In certain aspects, the camera 106, thecamera 107, and/or the camera 108 can recognize and detect depths andcolors of objects in the field of view 109, the field of view 110, andthe field of view 111, respectively. The camera 106, the camera 107,and/or the camera 108 can also provide other camera and video recorderfunctionalities, such as taking pictures, recording videos, streamingimages or other data, storing data in image buffers, etc. Thesefunctionalities may or may not include depth information. In connectionwith hardware and/or software processes consistent with the disclosedembodiments, the camera 106, the camera 107, and/or the camera 108 candetermine sizes, orientations, and visual properties of the one or moreobjects 104. The camera 106, the camera 107, and/or the camera 108 caninclude or embody any camera known to one of ordinary skill in the artcapable of handling the processes disclosed herein. The camera 106, thecamera 107, and/or the camera 108 can include appropriate hardware andsoftware components (e.g., circuitry, software instructions, etc.) fortransmitting signals and information to and from a pass/fail controller113 to conduct processes consistent with the disclosed embodiments. Thepass/fail controller can 113 comprise a computing device, a PLC, orother controller/processor. The camera 106, the camera 107, and/or thecamera 108 can output an image and/or one or more notifications to amonitor 114, a monitor 115, and a monitor 116, respectively.

Positioning of the one or more objects 104 into the field of view 109,the field of view 110, and the field of view 111 can occur at a start-upof the system 100 and can be adjusted during use of the system 100. Oneor more of the camera 106, the camera 107, and/or the camera 108 can beused to ensure proper positioning of the conveyor belt 101. For example,the camera 107 can be configured to generate an image of the area withinthe field of view 110. The camera 107 can determine a location of theone or more cleats 102 in the image. In an aspect, the camera 107 candetermine the location of the leading cleat. The camera 107 can comparethe determined location of the one or more cleats 102 in the image to areference location. If the determined location is equal to the referencelocation then no adjustment is necessary to the conveyor belt 101. Ifthe determined location is not equal to the reference location, thecamera 107 can determine an offset based on the difference between thedetermined location and the reference location. The offset can bedetermined in a measure of distance, for example, millimeters,centimeters, inches, and the like and/or the offset can be determined asa number of steps. The camera 107 can transmit a signal to the PLC 112to advance or reverse the conveyor belt 101 by the offset by engagingthe stepper motor 105.

In operation, the system 100 can be configured to assess a current stateof assembly of the one or more objects 104 and take one or more actionsbased on the current state of assembly. As each of the one or moreobjects 104 is advanced by the conveyor belt 101, the one or moreobjects 104 will each be placed in the field of view 109, the field ofview 110, and the field of view 111 of the camera 106, the camera 107,and/or the camera 108, respectively. While FIG. 1 illustrates only threecameras, it is specifically contemplated that less than three or morethan three cameras can be used. It is further contemplated that theconveyor belt 101 can be configured to have more than the illustratedthree objects 104 disposed thereon, regardless of the number of cameras.As the one or more objects 104 progress along the conveyor belt 101, oneor more items can be assembled into the one or more objects 104 by ahuman operator or a robot.

When each of the one or more objects 104 is within a field of view ofone of the cameras, the camera can generate an image of the object 104within the field of view associated with that camera. For example, thecamera 106 can generate an image of the area within the field of view109, the camera 107 can generate an image of the area within the fieldof view 110, and the camera 108 can generate an image of the area withinthe field of view 111. Each of the camera 106, the camera 107, and/orthe camera 108 can analyze their respective images. The analysis of animage can comprise determining a presence or absence of one or morepatterns. The one or more patterns can comprise a text pattern, anumeric pattern, a symbol pattern, and combinations thereof. Forexample, a text pattern can comprise any sequence of characters such as,“FILTER NEEDLE”. A numeric pattern can comprise any sequence of numberssuch as, “6941518”. The symbol pattern can comprise any sequence ofsymbols such as, “●□□♦”. In an aspect, the camera 106, the camera 107,and/or the camera 108 can utilize optical character recognition (OCR) to“read” the one or more patterns. In another aspect, the camera 106, thecamera 107, and/or the camera 108 can be configured to not utilize OCR,but rather can be configured to merely recognize the one or morepatterns as a specific pattern.

In an aspect, the one or more patterns can be embodied on the one ormore items to be assembled into the one or more objects 104. In anaspect, at least a portion of the one or more items can comprise one ormore associated patterns. Thus, in the event the camera 106, the camera107, and/or the camera 108 determines the presence of the one or morepatterns, the presence of the one or more patterns indicates a presenceof the item associated with a specific pattern. For example, if thecamera 106 determines the presence of “FILTER NEEDLE” in the image takenof the area within the field of view 109, then a conclusion can be drawnthat an item associated with the pattern “FILTER NEEDLE” is present inthe object 104 within the field of view 109. The camera 106, the camera107, and/or the camera 108 can be configured to determine the presenceor absence of a plurality of patterns within a single image. Forexample, the camera 106 can determine the presence of “FILTER NEEDLE”and “FILTER NEEDLE” in the image taken of the area within the field ofview 109. The presence of both patterns can indicate that an itemassociated with two occurrences of the pattern “FILTER NEEDLE” ispresent in the object 104 within the field of view 109.

Each of the items that can be assembled into the one or more objects 104can be associated with one or more patterns that indicate a presence orabsence of a specific number of the item. For example, an item may onlybe embodied with one occurrence of a specific pattern. If the camera106, the camera 107, and/or the camera 108 determine that the specificpattern only occurs once then the conclusion can be drawn that only oneof the item is present. However, if the camera 106, the camera 107,and/or the camera 108 determine that the specific pattern occurs two ormore times then the conclusion can be drawn that more than one of theitem is present. In another example, an item may be embodied with twooccurrences of a specific pattern. If the camera 106, the camera 107,and/or the camera 108 determine that the specific pattern only occurstwice then the conclusion can be drawn that only one of the item ispresent. However, if the camera 106, the camera 107, and/or the camera108 determine that the specific pattern occurs one or three or moretimes then the conclusion can be drawn that more than one of the item ispresent. In a further example, an item may be embodied with a range ofspecific patterns. For example, the item may be embodied with one to twooccurrences of the specific pattern. If the camera 106, the camera 107,and/or the camera 108 determine that the specific pattern occurs once ortwice then the conclusion can be drawn that only one of the item ispresent. However, if the camera 106, the camera 107, and/or the camera108 determine that the specific pattern occurs three or more times thenthe conclusion can be drawn that more than one of the item is present.

Each of the camera 106, the camera 107, and/or the camera 108 can beconfigured to analyze an entire image or one or more specific regions ofan image. FIG. 2 illustrates an example image 200 of an object 104. Theobject 104 can comprise a tray 201 configured for storing one or moreitems. The one or more items can be assembled into the tray 201 suchthat at least a portion of the one or more items is present in one ormore specific regions. The tray 201 can comprise one or more regions,for example, a region 202, a region 203, and a region 204. Each of theregion 202, the region 203, and the region 204 can be associated with anarea where the one or more patterns should be present if the item ispresent in the tray 201. For example, the region 202 can be associatedwith a location of a vial cap of a vial when assembled into the tray201, the region 203 can be associated with a location of one or moresyringes and/or one or more needles when assembled into the tray 201,and the region 204 can be associated with a location of one or morepamphlets when assembled into the tray 201. Each of the camera 106, thecamera 107, and/or the camera 108 can be configured to analyze one ormore assigned regions of the image 200. For example, the camera 106 canbe assigned to analyze the region 202 and the region 203, the camera 107can be assigned to analyze the region 203, and the camera 108 can beassigned to analyze the region 203 and the region 204. Any combinationof assigned regions is contemplated. Furthermore, each of the camera106, the camera 107, and/or the camera 108 can be configured todetermine presence or absence of one or more assigned patterns in theassigned regions. For example, the camera 106 can be assigned todetermine presence or absence of a vial cap in the region 202 andpresence or absence of a first pattern (including a number ofoccurrences of the first pattern) in the region 203, the camera 107 canbe assigned to determine presence or absence of a second pattern(including a number of occurrences of the second pattern) in the region203, and the camera 108 can be assigned to determine presence or absenceof a third pattern (including a number of occurrences of the thirdpattern) in the region 203 and presence or absence of a fourth pattern(including a number of occurrences of the fourth pattern) in the region204. Any combination of assigned patterns and assigned regions iscontemplated.

Returning to FIG. 1, each of the one or more objects 104 can beconfigured to contain a specific number of each of the one or moreitems. The presence of the specific number of each item indicates thatthe one or more objects 104 is correctly assembled. The presence ofanything other than the specific number of each item indicates that theone more objects 104 is incorrectly assembled. Each of the camera 106,the camera 107, and/or the camera 108 can be configured to make anindependent assessment of the object 104 within the respective field ofview. If a camera determines that the specific number of items thecamera is configured to detect is present, the camera can issue a PASSsignal to the pass/fail controller 113. If a camera determines that thespecific number of items the camera is configured to detect is notpresent, the camera can issue a FAIL signal to the pass/fail controller113. If each of the camera 106, the camera 107, and/or the camera 108issues a PASS signal to the pass/fail controller 113, then the pass/failcontroller 113 can provide a signal to the PLC 112 to cause the steppermotor 105 to advance the conveyor belt 101 to advance the one or moreobjects 104 to be positioned under the field of view of the next camera.The pass/fail controller 113 can further transmit a notification to eachof the monitors 114-116 to display a PASS notification. If one or moreof the camera 106, the camera 107, and/or the camera 108 issues a FAILsignal to the pass/fail controller 113, the pass/fail controller 113will not provide a signal to the PLC 112 to cause the stepper motor 105to advance. The pass/fail controller 113 can further transmit anotification to the monitors 114-116 associated with the camera(s)issuing the FAIL signal to display a FAIL notification. An operator(e.g., a human or a robot) positioned at the monitors 114-116 displayingthe FAIL notification can take corrective action to remedy the FAILstatus. For example, if the FAIL signal was issued as a result of amissing item, the operator can replace the missing item whereupon thecamera that made the prior FAIL determination can re-generate andre-analyze an image to determine that the item is now present and issuea PASS signal to the pass/fail controller 113. In another example, ifthe FAIL signal was issued as a result of one or more extra items, theoperator can remove the one or more extra items whereupon the camerathat made the prior FAIL determination can re-generate and re-analyze animage to determine that the required number of items is now present andissue a PASS signal to the pass/fail controller 113.

In a further aspect, the analysis of an image by the camera 106, thecamera 107, and/or the camera 108 can comprise not only determining thepresence of absence of the one or more patterns, but also determining arotation of two or more patterns. In an aspect, the two or more patternscan be embodied on the one or more items to be assembled into the one ormore objects 104 along a specific axis. In an aspect, at least a portionof the one or more items can comprise two or more associated patternsalong a specific axis. Thus, in the event the camera 106, the camera107, and/or the camera 108 determines the presence of the two or morepatterns along the specific axis, the presence of the two or morepatterns along the specific axis indicates a presence of the itemassociated with a specific pattern along the specific axis. For example,if the camera 106 determines the presence of “FILTER NEEDLE” and “FILTERNEEDLE” along the same axis (e.g., 30°, 60°, 90°, 120°, 180°, and thelike) in the image taken of the area within the field of view 109, thena conclusion can be drawn that an item associated with the pattern“FILTER NEEDLE” and “FILTER NEEDLE” along the same axis is present inthe object 104 within the field of view 109. The camera 106, the camera107, and/or the camera 108 can be configured to determine the rotationof a plurality of patterns within a single image. For example, thecamera 106 can determine the presence of “FILTER NEEDLE” and “FILTERNEEDLE” along a first axis and the presence of “SYRINGE NEEDLE” and“SYRINGE NEEDLE” along a second axis in the image taken of the areawithin the field of view 109. The presence of both patterns along twodifferent axes can indicate that an item associated with two occurrencesof the pattern “FILTER NEEDLE” along the first axis is present in theobject 104 and an item associated with two occurrences of the pattern“SYRINGE NEEDLE” along the second axis is also present in the object104. By way of further example, the camera 106 can determine thepresence of “FILTER NEEDLE” and “FILTER NEEDLE” along a first axis andthe presence of “FILTER NEEDLE” along a second axis in the image takenof the area within the field of view 109. The presence of both patternsalong two different axes can indicate that two occurrences of an itemassociated with the pattern “FILTER NEEDLE” are present in the object104.

Each of the items that can be assembled into the one or more objects 104can be associated with one or more patterns that are embodied along aspecific axis that indicate a presence or absence of a specific numberof the item. For example, an item may be embodied with two occurrencesof a specific pattern along a specific axis. If the camera 106, thecamera 107, and/or the camera 108 determine that the specific patternonly occurs twice along the specific axis then the conclusion can bedrawn that only one of the item is present. However, if the camera 106,the camera 107, and/or the camera 108 determine that the specificpattern occurs along more than one axis then the conclusion can be drawnthat more than one of the item is present.

FIG. 3A and FIG. 3B illustrate an example image 300 and 303 of a tray201 that comprises an item 301 and an item 302. The item 301 can be avial and the item 302 can be a filter needle, for example. Whichever ofthe camera 106, the camera 107, and/or the camera 108 that generates theimage 300 can determine that a vial cap is present in the region 202.The presence of a single vial cap indicates that the item 301 ispresent. The camera 106, the camera 107, and/or the camera 108 thatgenerates the image 300 can determine that, in the region 203, twooccurrences of a pattern are present, “TEXT A”. In an aspect, the twooccurrences of the pattern, “TEXT A”, can indicate that a one or morethan one instance of the item 302 is present and the camera 106, thecamera 107, and/or the camera 108 can generate a PASS or a FAIL signalas appropriate. In another aspect, depending on pattern configuration onthe item 302 (e.g., a single instance of the item 302 can have either asingle occurrence of “TEXT A” or a double occurrence of “TEXT A”) thecamera 106, the camera 107, and/or the camera 108 can determine whether“TEXT A” and “TEXT A” appear on the same axis. If “TEXT A” and “TEXT A”appear on the same axis then the camera 106, the camera 107, and/or thecamera 108 can determine that a single instance of the item 302 ispresent and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS or a FAIL signal as appropriate. If “TEXT A” and “TEXTA” appear on different axes then the camera 106, the camera 107, and/orthe camera 108 can determine that a more than one instance of the item302 is present and the camera 106, the camera 107, and/or the camera 108can generate a PASS or a FAIL signal as appropriate. In an aspect, thedetermination of axes can be used to confirm that any number of the item302 are present and generate a PASS or a FAIL signal based on theexpected number of instances of the item 302 versus the determinednumber of instances of the item 302.

FIG. 4A illustrates an example image 400 of the tray 201 that comprisesthe item 301 and two instances of the item 302. The camera 106, thecamera 107, and/or the camera 108 that generates the image 400 candetermine that, in the region 203, three occurrences of a pattern arepresent, (“TEXT A”). In an aspect, the three occurrences of the pattern,“TEXT A” can indicate that one or more than one instance of the item 302is present and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS or a FAIL signal as appropriate. In another aspect,depending on pattern configuration on the item 302 (e.g., a singleinstance of the item 302 can have a single occurrence of “TEXT A”, adouble occurrence of “TEXT A”, or a triple occurrence of “TEXT A”) thecamera 106, the camera 107, and/or the camera 108 can determine whetherthe three occurrences of “TEXT A” appear on the same axis. As shown inFIG. 4A, two occurrences of “TEXT A” appear on the same axis and oneoccurrence of “TEXT A” appears on a different axis. Accordingly, thecamera 106, the camera 107, and/or the camera 108 can determine thatmore than one instance of the item 302 is present and the camera 106,the camera 107, and/or the camera 108 can generate a PASS or a FAILsignal as appropriate. In an aspect, the determination of axes can beused to confirm that any number of the item 302 are present and generatea PASS or a FAIL signal based on the expected number of instances of theitem 302 versus the determined number of instances of the item 302.

FIG. 4B illustrates an example image 401 of the tray 201 that comprisesthe item 301, one instance of the item 302, and one instance of an item402. In one aspect, the camera 106, the camera 107, and/or the camera108 that generates the image 400 can determine that, in the region 203,two occurrences of a first pattern are present, (“TEXT A”) and oneoccurrence of a second pattern is present, (“TEXT B”). In an aspect, thetwo occurrences of the pattern, “TEXT A” can indicate that one or morethan one instance of the item 302 is present and the camera 106, thecamera 107, and/or the camera 108 can generate a PASS or a FAIL signalas appropriate. In another aspect, depending on pattern configuration onthe item 302 (e.g., a single instance of the item 302 can have a singleoccurrence of “TEXT A”, a double occurrence of “TEXT A”, or a tripleoccurrence of “TEXT A”) the camera 106, the camera 107, and/or thecamera 108 can determine whether the two occurrences of “TEXT A” appearon the same axis. As shown in FIG. 4B, the two occurrences of “TEXT A”appear on the same axis. Accordingly, the camera 106, the camera 107,and/or the camera 108 can determine that more one instance of the item302 is present and the camera 106, the camera 107, and/or the camera 108can generate a PASS or a FAIL signal as appropriate. However, the oneoccurrence of the pattern “TEXT B” can indicate that an item has beenplaced in the tray 201 that should not be in the tray 201 at this stagein the assembly process. Accordingly, the camera 106, the camera 107,and/or the camera 108 can generate a FAIL signal based on the presenceof a pattern that is not intended to be present.

In another aspect, the camera 106, the camera 107, and/or the camera 108that generates the image 400 can determine that the pattern “TEXT B” ispresent and can ignore the presence of the pattern “TEXT A” (or anyother pattern as required). In an aspect, the one occurrence of thepattern, “TEXT B” can indicate that one instance of the item 302 ispresent and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS signal.

FIG. 5A illustrates an example image 500 of the tray 201 that comprisesthe item 301, the item 302, and a single instance of an item 501. Thecamera 106, the camera 107, and/or the camera 108 that generates theimage 500 can be configured to ignore the vial cap in the region 202 andto ignore the presence of the pattern “TEXT A” in the region 203.Instead, the camera 106, the camera 107, and/or the camera 108 thatgenerates the image 400 can determine that, in the region 203, twooccurrences of another pattern are present, (“TEXT B”). In an aspect,the two occurrences of the pattern, “TEXT B” can indicate that eitherone or more than one instance of the item 501 is present and the camera106, the camera 107, and/or the camera 108 can generate a PASS or a FAILsignal as appropriate. In another aspect, depending on patternconfiguration on the item 501 (e.g., a single instance of the item 501can have a single occurrence of “TEXT B”, a double occurrence of “TEXTB”, or a triple occurrence of “TEXT B”) the camera 106, the camera 107,and/or the camera 108 can determine whether the two occurrences of “TEXTB” appear on the same axis. As shown in FIG. 5A, the two occurrences of“TEXT B” appear on the same axis. Accordingly, the camera 106, thecamera 107, and/or the camera 108 can determine that one instance of theitem 501 is present and the camera 106, the camera 107, and/or thecamera 108 can generate a PASS or a FAIL signal as appropriate. In anaspect, the determination of axes can be used to confirm that any numberof the item 501 are present and generate a PASS or a FAIL signal basedon the expected number of instances of the item 501 versus thedetermined number of instances of the item 501. FIG. 5B illustrates anexample image 503 of the tray 201 that comprises the item 301, the item302, and a single instance of the item 501. FIG. 5B is similar to FIG.5A with the exception that FIG. 5B illustrates that the pattern “TEXT B”occurs twice along the same axis, however at a different angle than theaxis in FIG. 5A.

FIG. 6A illustrates an example image 600 of the tray 201 that comprisesthe item 301 and two instances of the item 501. The camera 106, thecamera 107, and/or the camera 108 that generates the image 600 candetermine that, in the region 203, four occurrences of a pattern arepresent, “TEXT B”. In an aspect, the four occurrences of the pattern,“TEXT B” can indicate that one or more than one instance of the item 501is present and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS or a FAIL signal as appropriate. In another aspect,depending on pattern configuration on the item 501 (e.g., a singleinstance of the item 501 can have a single occurrence of “TEXT B”, adouble occurrence of “TEXT B”, a triple occurrence of “TEXT B”, or aquadruple occurrence of “TEXT B”) the camera 106, the camera 107, and/orthe camera 108 can determine the axes upon which the four occurrences of“TEXT B” appear. As shown in FIG. 5A, two occurrences of “TEXT B” appearon a first axis and the other two occurrences of “TEXT B” appear on asecond axis. Accordingly, as the two sets of “TEXT B” appear ondifferent axes, the camera 106, the camera 107, and/or the camera 108can determine that more than one instance of the item 501 is present andthe camera 106, the camera 107, and/or the camera 108 can generate aPASS or a FAIL signal as appropriate. In an aspect, the determination ofaxes can be used to confirm that any number of the item 501 are presentand generate a PASS or a FAIL signal based on the expected number ofinstances of the item 501 versus the determined number of instances ofthe item 501.

FIG. 6B illustrates an example image 601 of the tray 201 that comprisesthe item 301, the item 302, and two instances of the item 501. Thecamera 106, the camera 107, and/or the camera 108 that generates theimage 601 can determine that, in the region 203, three occurrences of apattern are present, “TEXT B”. In an aspect, the three occurrences ofthe pattern, “TEXT B” can indicate that one or more than one instance ofthe item 501 is present and the camera 106, the camera 107, and/or thecamera 108 can generate a PASS or a FAIL signal as appropriate. Inanother aspect, depending on pattern configuration on the item 501(e.g., a single instance of the item 501 can have a single occurrence of“TEXT B”, a double occurrence of “TEXT B”, a triple occurrence of “TEXTB”, or a quadruple occurrence of “TEXT B”) the camera 106, the camera107, and/or the camera 108 can determine the axes upon which the threeoccurrences of “TEXT B” appear. As shown in FIG. 6B, two occurrences of“TEXT B” appear on a first axis and the one occurrence of “TEXT B”appears on a second axis. Accordingly, as the two sets of “TEXT B”appear on different axes, the camera 106, the camera 107, and/or thecamera 108 can determine that more than one instance of the item 501 ispresent and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS or a FAIL signal as appropriate. In an aspect, thedetermination of axes can be used to confirm that any number of the item501 are present and generate a PASS or a FAIL signal based on theexpected number of instances of the item 501 versus the determinednumber of instances of the item 501.

FIG. 7A illustrates an example image 700 of the tray 201 that comprisesthe item 301, the item 302, the item 501, a single instance of an item701, and a single instance of an item 702. The camera 106, the camera107, and/or the camera 108 that generates the image 700 can beconfigured to ignore the vial cap in the region 202 and to ignore thepresence of the patterns “TEXT A” and “TEXT B” in the region 203.Instead, the camera 106, the camera 107, and/or the camera 108 thatgenerates the image 700 can determine that, in the region 203, twooccurrences of another pattern are present, (“TEXT D”). In an aspect,the two occurrences of the pattern, “TEXT D” can indicate that eitherone or more than one instance of the item 701 is present and the camera106, the camera 107, and/or the camera 108 can generate a PASS or a FAILsignal as appropriate. In another aspect, depending on patternconfiguration on the item 701 (e.g., a single instance of the item 701can have a single occurrence of “TEXT D”, a double occurrence of “TEXTD”, or a triple occurrence of “TEXT D”) the camera 106, the camera 107,and/or the camera 108 can determine whether the two occurrences of “TEXTD” appear on the same axis. As shown in FIG. 7A, the two occurrences of“TEXT D” appear on the same axis. Accordingly, the camera 106, thecamera 107, and/or the camera 108 can determine that one instance of theitem 701 is present and the camera 106, the camera 107, and/or thecamera 108 can generate a PASS or a FAIL signal as appropriate. In anaspect, the determination of axes can be used to confirm that any numberof the item 701 are present and generate a PASS or a FAIL signal basedon the expected number of instances of the item 701 versus thedetermined number of instances of the item 701. In the same image 700,the camera 106, the camera 107, and/or the camera 108 can determinethat, in the region 204, two occurrences of another pattern are present,(“TEXT C”). In an aspect, the two occurrences of the pattern, “TEXT C”can indicate that either one or more than one instance of the item 702is present and the camera 106, the camera 107, and/or the camera 108 cangenerate a PASS or a FAIL signal as appropriate. In another aspect,depending on pattern configuration on the item 702 (e.g., a singleinstance of the item 702 can have a single occurrence of “TEXT C”, adouble occurrence of “TEXT C”, or a triple occurrence of “TEXT C”) thecamera 106, the camera 107, and/or the camera 108 can determine whetherthe two occurrences of “TEXT C” appear on the same axis. As shown inFIG. 7A, the two occurrences of “TEXT C” appear on the same axis.Accordingly, the camera 106, the camera 107, and/or the camera 108 candetermine that one instance of the item 702 is present and the camera106, the camera 107, and/or the camera 108 can generate a PASS or a FAILsignal as appropriate. In an aspect, the determination of axes can beused to confirm that any number of the item 702 are present and generatea PASS or a FAIL signal based on the expected number of instances of theitem 702 versus the determined number of instances of the item 702. FIG.7B illustrates an example image 703 of the tray 201 that comprises theitem 301, the item 302, the item 501, a single instance of the item 701,and a single instance of the item 702. FIG. 7B is similar to FIG. 7Awith the exception that FIG. 7B illustrates that the pattern “TEXT D”occurs twice along the same axis, however at a different angle than theaxis in FIG. 7A and similarly the pattern “TEXT C” occurs twice alongthe same axis, however at a different angle than the axis in FIG. 7A.

FIG. 8A illustrates an example image 800 of the tray 201 that comprisesthe item 301, the item 302, the item 501, two instances of the item 701,and a single instance of the item 702. The camera 106, the camera 107,and/or the camera 108 that generates the image 800 can determine that,in the region 203, three occurrences of a pattern are present, “TEXT D”.In an aspect, the three occurrences of the pattern, “TEXT D” canindicate that one or more than one instance of the item 701 is presentand the camera 106, the camera 107, and/or the camera 108 can generate aPASS or a FAIL signal as appropriate. In another aspect, depending onpattern configuration on the item 701 (e.g., a single instance of theitem 701 can have a single occurrence of “TEXT D”, a double occurrenceof “TEXT D”, a triple occurrence of “TEXT D”, or a quadruple occurrenceof “TEXT D”) the camera 106, the camera 107, and/or the camera 108 candetermine the axes upon which the three occurrences of “TEXT D” appear.As shown in FIG. 8A, two occurrences of “TEXT D” appear on a first axisand the one occurrence of “TEXT D” appears on a second axis.Accordingly, as the two sets of “TEXT D” appear on different axes, thecamera 106, the camera 107, and/or the camera 108 can determine thatmore than one instance of the item 701 is present and the camera 106,the camera 107, and/or the camera 108 can generate a PASS or a FAILsignal as appropriate. In an aspect, the determination of axes can beused to confirm that any number of the item 701 are present and generatea PASS or a FAIL signal based on the expected number of instances of theitem 701 versus the determined number of instances of the item 701. FIG.8B is similar to FIG. 8A with the exception that FIG. 8B illustratesthat the pattern “TEXT D” occurs twice along a first axis and once alonga second axis, however the first and second axes are at different anglesthan the axes in FIG. 8A.

Returning to FIG. 1, each of the camera 106, the camera 107, and thecamera 108 can independently determine both the presence or the absenceof one or more patterns in an image and determine a rotation of each theone or more patterns in the image of an object 104. Each of the camera106, the camera 107, and the camera 108 can perform an action based onthe presence or absence and the rotation of the one or more patterns inthe image. If a camera determines that a correct number of an item ispresent in the image of an object 104 based on presence of pattern androtation of the pattern, the action can comprise transmitting a PASSsignal to the pass/fail controller 113. If the camera determines that anincorrect number of an item is present in the image of an object 104based on presence of pattern and rotation of the pattern, the action cancomprise transmitting a FAIL signal to the pass/fail controller 113. Ifeach of the camera 106, the camera 107, and/or the camera 108 issues aPASS signal to the pass/fail controller 113, then the pass/failcontroller 113 can provide a signal to the PLC 112 to cause the steppermotor 105 to advance the conveyor belt 101 to advance the one or moreobjects 104 to be positioned under the field of view of the next camera.The pass/fail controller 113 can further transmit a notification to eachof the monitors 114-116 to display a PASS notification. If one or moreof the camera 106, the camera 107, and/or the camera 108 issues a FAILsignal to the pass/fail controller 113, the pass/fail controller 113will not provide a signal to the PLC 112 to cause the stepper motor 105to advance. The pass/fail controller 113 can further transmit anotification to the monitors 114-116 associated with the camera(s)issuing the FAIL signal to display a FAIL notification. An operator(e.g., a human or a robot) positioned at the monitors 114-116 displayingthe FAIL notification can take corrective action to remedy the FAILstatus.

In another aspect, one or more of the camera 106, the camera 107, andthe camera 108 can count a number of the one or more objects 104. Forexample, a the one or more objects 104 pass by one of the camera 106,the camera 107, and the camera 108, the camera can increment a tally ofthe one or more objects 104 imaged by the camera. In a further aspect, anumber of empty locations can be interspersed between the one or moreobjects 104. For example, in certain scenarios one or more of the camera106, the camera 107, and the camera 108 may not have an object 104within a respective field of view. The conveyor belt 101 can have apattern (e.g., a “no tray” pattern) embodied thereon in a position wherethe object 104 would otherwise be placed. The camera 106, the camera107, and the camera 108 can identify the pattern and issue a PASS signalto contribute to advancement of the conveyor belt 101.

FIG. 9 illustrates an example embodiment of the system 100 illustratingpositioning of camera 106, the camera 107, and the camera 108 relativeto the conveyor belt 101. FIG. 9 further illustrates positioning of themonitors 114-116. The stepper motor 105 is illustrated at one end of theconveyor belt 101. One or more of the PLC 112 and/or the pass/failcontroller 113 can be contained with a housing 901. One or moredispensers 902 can be configured for storing one or more items to beaccessed during assembly into the one or more objects 104. The system100 can comprise one or more emergency stop (“E-Stop”) buttons 903. TheE-Stop buttons 903 can be engaged at any point in time to temporarilycease operation of the system 100, for any reason. The E-Stop buttons903 can be reset, and the system 100 restarted (e.g., by an operator ortechnician that has determined that it is safe to do so). The system 100can comprise one or more OptoSwitches 904. The OptoSwitches 904 can beactuated (“tripped”) by placing a finger or thumb in the saddle-likestructure of the OptoSwitch 904. This action breaks an optical signalpath, causing a switch condition. The OptoSwitches 904 can be used toaccept a visual inspection during “Manual Trigger” mode, andstart/restart the belt motion during “Autonomous” (or “Auto”) mode.

The system 100 can comprise a key switch mechanism 905. The key switchmechanism 905 can be used to toggle between an “Autonomous” Mode and“Manual Trigger” Mode. Under normal operation, regardless of mode, afirst operator station can comprise an operator loading trays onto theconveyor belt 101. In an aspect, these trays can be pre-fitted with aprefilled capped vial. In manual trigger mode, at a second operatorstation, an operator can load a filter needle tip into the tray. Afterthis operation, the camera 106 inspects the tray for the appropriateitems. At a third operator station, an injection needle tip can be addedto the tray. Then, the camera 107 inspects the tray for appropriateitems. At a fourth operator station, an operator loads an emptyblister-packed syringe into the tray. Afterwards, a fifth operator loadsa Physician Insert (PI) into the tray. After the PI is loaded, thecamera 108 inspects the tray for completed loading. Once the tray passesthis last station, the fully populated tray exits the conveyor belt 101for boxing.

In automated mode, trays are moved down the conveyor belt 101automatically. The system 100 can maintain a dwell time (e.g., 1-5seconds) before the conveyor belt 101 shifts to the next position. Theshift occurs only when all three inspection cameras (e.g., the camera106, the camera 107, and the camera 108) clear the tray (“Pass”) that isbeing inspected by a respective camera. An issue at any inspectionstation can result in a the conveyor belt 101 and a “red light”condition (“Fail”), at which point an operator can correct the issue orpull the tray from the conveyor belt 101 (each camera can allow theconveyor belt 101 to advance when there is no tray in its field ofview). The advancement of the conveyor belt 101 can be dependent on allcameras detecting a “passing” tray configuration. A display screen(e.g., the monitors 114-116) at each camera station can display theassociated camera's video stream, with overlaid “Pass”, “Fail”, or “NoJob” statuses depending on the inspection results. Camera online statuscan be reset from the monitors 114-116 if required during operation.

In an aspect, illustrated in FIG. 10, a method 1000 is disclosedcomprising obtaining a first image of a tray at 1010. The method 1000can comprise determining a presence or absence of one or more firstpatterns in the first image at 1020. The one or more first patterns cancomprise text patterns, numeric patterns, symbol patterns, andcombinations thereof. The method 1000 can comprise determining arotation of each the one or more first patterns in the first image at1030. The method 1000 can comprise performing an action based on thepresence or absence and the rotation of the one or more first patternsin the first image at 1040. In an aspect, each step of the method 1000can be performed by a computing device, a camera (with processingfunctionality), or a combination thereof. In some aspect, multiplecomputing devices and/or cameras can be employed to perform the method1000. For example, multiple cameras can be used wherein a first cameracan perform steps 1010, 1020, and step 1030 while a second cameraperforms step 1040. In another aspect, the method 1000 can be repeatedat each of several cameras and/or computing devices as a tray proceedsalong an assembly line. For example, steps 1010, 1020, 1030, and 1040can be performed by a first camera for a specific pattern(s), then steps1010, 1020, 1030, and 1040 can be performed again by a second camera foranother specific pattern(s). Still further, one or more sub-stepsdescribed herein can be performed by a designated camera and/orcomputing device.

Determining a presence or absence of one or more first patterns in thefirst image can comprise determining presence of one or two of the oneor more first patterns and wherein determining a rotation of each theone or more first patterns in the first image can comprise determiningthat the one or two of the one or more first patterns are on a firstaxis. Performing an action based on the presence or absence and therotation of the one or more first patterns in the first image cancomprise generating a pass inspection signal and advancing a belt havingthe tray disposed thereon. Determining a presence or absence of one ormore first patterns in the first image can comprise determining presenceof three or more of the one or more first patterns. Performing an actionbased on the presence or absence and the rotation of the one or morefirst patterns in the first image can comprise generating a failinspection signal and notifying an operator that a first item associatedwith the one or more first patterns should be removed from the tray.Determining a presence or absence of one or more first patterns in thefirst image can comprise determining presence of two of the one or morefirst patterns and wherein determining a rotation of each the one ormore first patterns in the first image can comprise determining that thetwo of the one or more first patterns are not on a same axis. Performingan action based on the presence or absence and the rotation of the oneor more first patterns in the first image can comprise generating a failinspection signal and notifying an operator that a first item associatedwith the one or more first patterns should be removed from the tray.

The method 1000 can further comprise obtaining a second image of thetray, determining a presence or absence of one or more second patternsin the second image, determining a rotation of each the one or moresecond patterns in the second image, and performing an action based onthe presence or absence and the rotation of the one or more secondpatterns in the second image. The one or more second patterns cancomprise text patterns, numeric patterns, symbol patterns, andcombinations thereof. Determining a presence or absence of one or moresecond patterns in the second image can comprise determining presence ofone or two of the one or more second patterns and wherein determining arotation of each the one or more second patterns in the second image cancomprise determining that the one or two of the one or more secondpatterns are on a second axis. Performing an action based on thepresence or absence and the rotation of the one or more second patternsin the second image can comprise generating a pass inspection signal andadvancing a belt having the tray disposed thereon. Determining apresence or absence of one or more second patterns in the second imagecan comprise determining presence of three or more of the one or moresecond patterns. Performing an action based on the presence or absenceand the rotation of the one or more second patterns in the second imagecan comprise generating a fail inspection signal and notifying anoperator that a second item associated with the one or more secondpatterns should be removed from the tray. Determining a presence orabsence of one or more second patterns in the second image can comprisedetermining presence of two of the one or more second patterns andwherein determining a rotation of each the one or more second patternsin the second image can comprise determining that the two of the one ormore second patterns are not on a same axis. Performing an action basedon the presence or absence and the rotation of the one or more secondpatterns in the second image can comprise generating a fail inspectionsignal and notifying an operator that a second item associated with theone or more second patterns should be removed from the tray.

The method 1000 can further comprise determining a location of a cleatin the first image, comparing the determined location of the cleat inthe first image to a reference location, determining that the determinedlocation is different from the reference location, determining an offsetbased on the difference between the determined location and thereference location, and transmitting a signal to a belt controller toadjust a distance to advance a belt having the tray disposed thereon bythe offset. The offset can be one of a negative value, a positive value,or a zero value. In an aspect, determining the offset based on thedifference between the determined location and the reference location,and transmitting the signal to the belt controller to adjust thedistance to advance the belt having the tray disposed thereon by theoffset can be performed by one or more cameras. For example, a singlecamera can be designated to determine the offset. The offsetdetermination can be made after each movement of the belt.

The method 1000 can further comprise repeatedly obtaining a first imageof a tray, determining a presence or absence of one or more firstpatterns in the first image, determining a rotation of each the one ormore first patterns in the first image, and performing an action basedon the presence or absence and the rotation of the one or more firstpatterns in the first image for each of a plurality of trays.

The method 1000 can further comprise counting a number of the pluralityof trays, wherein a number of empty tray locations are interspersedbetween the plurality of trays. The method 1000 can further comprisecounting a number of the empty tray locations. Determining the presenceor absence of one or more first patterns in the first image can comprisedetermining a no tray pattern. Performing the action based on thepresence or absence and the rotation of the one or more second patternsin the first image can comprise advancing a belt having the no traypattern disposed thereon.

In an exemplary aspect, the methods and systems can be implemented on acomputer 1101 as illustrated in FIG. 11 and described below. By way ofexample, the camera 106, the camera 107, the camera 108, the PLC 112,and/or the pass/fail controller 113 (or a component thereof) of FIG. 1can be a computer 1101 as illustrated in FIG. 11. Similarly, the methodsand systems disclosed can utilize one or more computers to perform oneor more functions in one or more locations. FIG. 2 is a block diagramillustrating an exemplary operating environment 1100 for performing thedisclosed methods. This exemplary operating environment 1100 is only anexample of an operating environment and is not intended to suggest anylimitation as to the scope of use or functionality of operatingenvironment architecture. Neither should the operating environment 1100be interpreted as having any dependency or requirement relating to anyone or combination of components illustrated in the exemplary operatingenvironment 1100.

The present methods and systems can be operational with numerous othergeneral purpose or special purpose computing system environments orconfigurations. Examples of well known computing systems, environments,and/or configurations that can be suitable for use with the systems andmethods comprise, but are not limited to, personal computers, servercomputers, laptop devices, and multiprocessor systems. Additionalexamples comprise set top boxes, programmable consumer electronics,network PCs, programmable logic controllers (PLCs), minicomputers,mainframe computers, distributed computing environments that compriseany of the above systems or devices, and the like.

The processing of the disclosed methods and systems can be performed bysoftware components. The disclosed systems and methods can be describedin the general context of computer-executable instructions, such asprogram modules, being executed by one or more computers or otherdevices. Generally, program modules comprise computer code, routines,programs, objects, components, data structures, and/or the like thatperform particular tasks or implement particular abstract data types.The disclosed methods can also be practiced in grid-based anddistributed computing environments where tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules can be located inlocal and/or remote computer storage media including memory storagedevices.

Further, one skilled in the art will appreciate that the systems andmethods disclosed herein can be implemented via a general-purposecomputing device in the form of a computer 1101. The computer 1101 cancomprise one or more components, such as one or more processors 1103, asystem memory 1112, and a bus 1113 that couples various components ofthe computer 1101 including the one or more processors 1103 to thesystem memory 1112. In the case of multiple processors 1103, the systemcan utilize parallel computing.

The bus 1113 can comprise one or more of several possible types of busstructures, such as a memory bus, memory controller, a peripheral bus,an accelerated graphics port, and a processor or local bus using any ofa variety of bus architectures. The bus 1113, and all buses specified inthis description can also be implemented over a wired or wirelessnetwork connection.

The computer 1101 typically comprises a variety of computer readablemedia. Exemplary readable media can be any available media that isaccessible by the computer 1101 and comprises, for example and not meantto be limiting, both volatile and non-volatile media, removable andnon-removable media. The system memory 1112 can comprise computerreadable media in the form of volatile memory, such as random accessmemory (RAM), and/or non-volatile memory, such as read only memory(ROM). The system memory 1112 typically can comprise data such as imageanalysis data 1107 and/or program modules such as operating system 1105and image analysis software 1106 that are accessible to and/or areoperated on by the one or more processors 1103.

In another aspect, the computer 1101 can also comprise otherremovable/non-removable, volatile/non-volatile computer storage media.The mass storage device 1104 can provide non-volatile storage ofcomputer code, computer readable instructions, data structures, programmodules, and other data for the computer 1101. For example, a massstorage device 1104 can be a hard disk, a removable magnetic disk, aremovable optical disk, magnetic cassettes or other magnetic storagedevices, flash memory cards, CD-ROM, digital versatile disks (DVD) orother optical storage, random access memories (RAM), read only memories(ROM), electrically erasable programmable read-only memory (EEPROM), andthe like.

Optionally, any number of program modules can be stored on the massstorage device 1104, including by way of example, an operating system1105 and image analysis software 1106. One or more of the operatingsystem 1105 and image analysis software 1106 (or some combinationthereof) can comprise elements of the programming and the image analysissoftware 1106. Image analysis data 1107 can also be stored on the massstorage device 1104. Image analysis data 1107 can be stored in any ofone or more databases known in the art. Examples of such databasescomprise, DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®,mySQL, PostgreSQL, and the like. The databases can be centralized ordistributed across multiple locations within the network 1115.

In another aspect, the user can enter commands and information into thecomputer 1101 via an input device (not shown). Examples of such inputdevices comprise, but are not limited to, a keyboard, pointing device(e.g., a computer mouse, remote control), a microphone, a joystick, ascanner, touch-enabled devices such as a touchscreen, tactile inputdevices such as gloves and other body coverings, motion sensors, and thelike. These and other input devices can be connected to the one or moreprocessors 1103 via a human machine interface 1102 that is coupled tothe bus 1113, but can be connected by other interface and busstructures, such as, but not limited to, a parallel port, game port, anIEEE 1394 Port (also known as a Firewire port), a serial port, networkadapter 1108, and/or a universal serial bus (USB).

In yet another aspect, a display device 1111 can also be connected tothe bus 1113 via an interface, such as a display adapter 1109. It iscontemplated that the computer 1101 can have more than one displayadapter 1109 and the computer 1101 can have more than one display device1111. For example, a display device 1111 can be a monitor, an LCD(Liquid Crystal Display), light emitting diode (LED) display,television, smart lens, smart glass, and/or a projector. In addition tothe display device 1111, other output peripheral devices can comprisecomponents such as speakers (not shown) and a printer (not shown) whichcan be connected to the computer 1101 via Input/Output Interface 1110.Any step and/or result of the methods can be output in any form to anoutput device. Such output can be any form of visual representation,including, but not limited to, textual, graphical, animation, audio,tactile, and the like. The display 1111 and computer 1101 can be part ofone device, or separate devices.

In an aspect, the computer 1101 can be coupled to the system 100 via theInput/Output Interface 1110. The computer 1101 can be configured tomonitor and store data. The computer 1101 can be configured to storeimages acquired by cameras connected to the system 100, store datarelated to pass/fail statistics generated during system-generatedinspections, etc. The computer 1101 can also be used as a programminginterface to one or more smart devices (e.g., smart cameras) and/orembedded logic controllers that require customized firmware to operate.The computer 1101 can be used to generate, troubleshoot, upload, andstore iterations of this software or firmware.

The computer 1101 can operate in a networked environment using logicalconnections to one or more remote computing devices 1114 a,b,c. By wayof example, a remote computing device 1114 a,b,c can be a personalcomputer, computing station (e.g., workstation), portable computer(e.g., laptop, mobile phone, tablet device), smart device (e.g.,smartphone, smart watch, activity tracker, smart apparel, smartaccessory), security and/or monitoring device, a server, a router, anetwork computer, a peer device, edge device or other common networknode, and so on. Logical connections between the computer 1101 and aremote computing device 1114 a,b,c can be made via a network 1115, suchas a local area network (LAN) and/or a general wide area network (WAN).Such network connections can be through a network adapter 1108. Anetwork adapter 1108 can be implemented in both wired and wirelessenvironments. Such networking environments are conventional andcommonplace in dwellings, offices, enterprise-wide computer networks,intranets, and the Internet. In an aspect, the network adapter 1108 canbe configured to provide power to one or more connected devices (e.g., acamera). For example, the network adapter 1108 can adhere to thePower-over-Ethernet (PoE) standard or the like.

For purposes of illustration, application programs and other executableprogram components such as the operating system 1105 are illustratedherein as discrete blocks, although it is recognized that such programsand components can reside at various times in different storagecomponents of the computing device 1101, and are executed by the one ormore processors 1103 of the computer 1101. An implementation of imageanalysis software 1106 can be stored on or transmitted across some formof computer readable media. Any of the disclosed methods can beperformed by computer readable instructions embodied on computerreadable media. Computer readable media can be any available media thatcan be accessed by a computer. By way of example and not meant to belimiting, computer readable media can comprise “computer storage media”and “communications media.” “Computer storage media” can comprisevolatile and non-volatile, removable and non-removable media implementedin any methods or technology for storage of information such as computerreadable instructions, data structures, program modules, or other data.Exemplary computer storage media can comprise RAM, ROM, EEPROM, flashmemory or other memory technology, CD-ROM, digital versatile disks (DVD)or other optical storage, magnetic cassettes, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and which can beaccessed by a computer.

The methods and systems can employ artificial intelligence (AI)techniques such as machine learning and iterative learning. Examples ofsuch techniques include, but are not limited to, expert systems, casebased reasoning, Bayesian networks, behavior based AI, neural networks,fuzzy systems, evolutionary computation (e.g. genetic algorithms), swarmintelligence (e.g. ant algorithms), and hybrid intelligent systems (e.g.Expert inference rules generated through a neural network or productionrules from statistical learning).

The disclosed methods and systems were implemented, tested, and resultscompared with a standard manual-only, operator-driven assembly lineprocess. The following table indicates the disclosed methods and systemsoutperform the standard manual-only, operator-driven assembly lineprocess:

Assembly Line with Standard, Integrated Manual-Only Electronic VisualProcess Inspection Difference Line Rate 11 20 9 (units/min) Operators on25 22 (3.00) Line (people) Labor cost per     $0.46     $0.23    ($0.23) carton Labor Cost @  $10,633.03  $5,343.44  ($5,289.59)$12/hour Overhead @    $106.33    $60.72    ($45.61) $3/hour Cost perLot  $10,739.36  $5,404.16  ($5,335.20) Cost per year $385,048.98$192,502.86 ($192,546.12) (Based on 837,063 units in a year)

While the methods and systems have been described in connection withpreferred embodiments and specific examples, it is not intended that thescope be limited to the particular embodiments set forth, as theembodiments herein are intended in all respects to be illustrativerather than restrictive.

Unless otherwise expressly stated, it is in no way intended that anymethod set forth herein be construed as requiring that its steps beperformed in a specific order. Accordingly, where a method claim doesnot actually recite an order to be followed by its steps or it is nototherwise specifically stated in the claims or descriptions that thesteps are to be limited to a specific order, it is no way intended thatan order be inferred, in any respect. This holds for any possiblenon-express basis for interpretation, including: matters of logic withrespect to arrangement of steps or operational flow; plain meaningderived from grammatical organization or punctuation; the number or typeof embodiments described in the specification.

It will be apparent to those skilled in the art that variousmodifications and variations can be made without departing from thescope or spirit. Other embodiments will be apparent to those skilled inthe art from consideration of the specification and practice disclosedherein. It is intended that the specification and examples be consideredas exemplary only, with a true scope and spirit being indicated by thefollowing claims.

What is claimed is:
 1. A system comprising: a belt; a plurality ofimagers, each configured to: obtain an image of its respective field ofview, analyze the image to determine a quantity and/or orientation ofone or more items within its respective field of view, and generate apass inspection signal or a fail inspection signal, based on thequantity and/or orientation of the one or more items within itsrespective field of view; wherein at least one of the plurality ofimagers is further configured to count a number of a plurality of traysdisposed on the belt, wherein a number of empty tray locations areinterspersed between the plurality of trays; and a processor, coupled toeach of the plurality of imagers, configured to, receive the passinspection signal or the fail inspection signal, and advance the beltbased on receiving a pass inspection signal from each of the pluralityof imagers.
 2. The system of claim 1, wherein the one or more itemscomprise one or more first patterns.
 3. The system of claim 2, whereinthe one or more first patterns comprise text patterns, numeric patterns,symbol patterns, or combinations thereof.
 4. The system of claim 2,wherein, to determine a quantity and/or orientation of the one or moreitems, at least one of the plurality of imagers is further configuredto: determine presence of two of the one or more first patterns withinits respective field of view; and determine that the two of the one ormore first patterns are not aligned on a same axis within its respectivefield of view.
 5. The system of claim 1, wherein at least one of theplurality of imagers is further configured to count a number of theempty tray locations.
 6. The system of claim 2, wherein at least one ofthe plurality of imagers is further configured to determine that the oneor more first patterns in the image comprises a no tray pattern.
 7. Thesystem of claim 1, wherein the processor is configured to advance thebelt only after receiving a pass inspection signal from each of theplurality of imagers.
 8. The system of claim 1, wherein each of theplurality of imagers is configured to generate a pass inspection signalor a fail inspection signal, based on the quantity and/or orientation ofthe one or more items determined while the belt is stationary.
 9. Thesystem of claim 1, wherein the field of view of each of the plurality ofimagers, at any given time, encompasses different portions of the beltthan is present in any other field of view.
 10. The system of claim 3,wherein at least one of the imagers is configured to: determine whethermultiple instances of the text patterns, numeric patterns, symbolpatterns, or combinations thereof, are aligned along a same axis, andgenerate the pass inspection signal or fail inspection signal based onthe determination.
 11. The system of claim 1, wherein the one or moreitems includes one or more of a syringe, a syringe needle, or anauto-injector.
 12. The system of claim 1, wherein the processor isconfigured to withhold sending a signal that advances the belt based onreceiving a fail inspection signal from at least one of the plurality ofimagers.
 13. A system comprising: a belt; a plurality of imagers, eachconfigured to: obtain an image of its respective field of view, analyzethe image to determine a quantity and/or orientation of one or moreitems within its respective field of view, and generate a passinspection signal or a fail inspection signal, based on the quantityand/or orientation of the one or more items within its respective fieldof view; and a processor, coupled to each of the plurality of imagers,configured to, receive the pass inspection signal or the fail inspectionsignal, and advance the belt based on receiving a pass inspection signalfrom each of the plurality of imagers; wherein at least one of theplurality of imagers is further configured to: determine a location of acleat within its respective field of view; compare the determinedlocation of the cleat within its respective field of view to a referencelocation; determine that the determined location is different from thereference location; determine an offset based on the difference betweenthe determined location and the reference location; and transmit asignal to the processor to adjust a distance to move the belt by theoffset.
 14. The system of claim 13, wherein: the belt includes aplurality of cleats spaced apart from one another; a length of eachcleat is substantially perpendicular to a length of the belt; and one ormore of the fields of view are located between adjacent cleats of theplurality of cleats.